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Abstract 

A fundamental problem in debugging and monitoring is detecting 
whether the state of a system satisfies some predicate. If the system 
is distributed, then the resulting uncertainty in the state of the sys- 
tem makes such detection, in general, ill-defined. This paper presents 
three algorithms for detecting global predicates in a well-defined way. 

These algorithms do so by interpreting predicates with respect to the 
communication that has occurred in the system. 

A fundamental problem in debugging and monitoring is detecting whether 
the state of a system satisfies some predicate. If the system is distributed, 
then the resulting uncertainty in the state of the system makes such de- 
tection, in general, ill-defined. This paper presents three algorithms for 
detecting global predicates in a well-defined way. These algorithms do so by 
interpreting predicates with respect to the communication that has occurred 
in the system. Briefly, the first algorithm determines that the predicate was 
possibly true at some point in the past; the second determines that the pred- 
icate was definitely true in the past; while the third algorithm establishes 
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that the predicate is currently true, but to do so it may delay the execution 
of certain processes. 

Our approach is in contrast to the considerable body of work that uses 
temporal predicates (i.e., predicates expressed over process histories) for dis- 
tributed monitoring. Temporal predicates are more powerful, but also more 
complex to use. In many cases, the condition that the programmer wishes 
to monitor is simply and intuitively viewed as a predicate over the “in- 
stantaneous” state of the system. Using the possibly/definitely /currently 
interpretation such a pr^icate becomes well-defined, without requiring it 
to be recast using temporal formulas. Further, our algorithms may be more 
efficient than techniques that use a notion of explicit time or process histo- 
ries. Section 1 specifies the protocols and Section 2 gives an outline of their 
operation. 

This work arose as part of Meta, a toolkit supporting distributed system 
monitoring and control. The architecture addressed by Meta is more general 
than that needed for debugging, in that we are also concerned with several 
monitoring components reacting in a consistent and fault- tolerant manner. 
Section 3 discusses how the algorithms of this paper can be used to provide 
a breakpoint and tracepoint facility for Meta. 

1 Uncertainty in Distributed Systems 

Suppose we have a set of n processes P = {pi , pj, . . . , p n } that communicate 
only by message passing, and we have a monitoring process po that wishes 
to determine if the state of P satisfies a global state property In general, 
the uncertainty in the state of the system makes such determination impos- 
sible. For example, if each process has an integer- valued variable x and $ is 
“the median value of x is greater than 10 w , then po would have to know the 
simultaneous value of at least half of the variables to determine whether $ 
holds. 

Following are two ways that $ can be changed so that it is meaningful 
in the face of uncertainty: 
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GP1: There is an execution of P consistent with its observed behavior 
such that $ was true at a point in that execution. We will refer to 
this property as possibly 

GP2: For all executions of P consistent with its observed behavior, $ was 
true at some point. We will refer to this property as definitely 

Both of these conditions refer to some past state or states of P; a third 
property can be detected if it is acceptable to temporarily block processes: 

GPS: $ currently holds, and there is a logical execution of the unblocked 
system such that $ holds. We will refer to this property as currently 

Our protocols are based on each process in P informing po when a lo- 
cal event changes some local state on which $ depends. We say that such 
events potentially change Then, po constructs the partial order of events, 
represented compactly as vectors of timestamps [7], in order to determine 
whether possibly $ or definitely $ held. The first two conditions are illus- 
trated in Figure 1. In this space-time diagram, three processes Pi,P 2 ,P 3 
have variables respectively. Possibly (a?x > 10) A (s 2 > 10) and 

definitely (x* > 10) A > 10) hold in this execution. 





Figure 1: J = (at least two of the three x variables are greater than 10). 
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These three specifications provide different ways to manage the uncer- 
tainty of debugging in a distributed environment. The distinction between 
possibly and definitely arises from the relative nature of time in a distributed 
system [4]. To an omniscient observer with a global clock, two events that 
we regard as concurrent (i.e. a /* b A b /* a) will have some total order. 
We cannot determine this total order in any practical way, but determining 
that some predicate was possibly true provides us with valuable information 
about the behavior of the program. For instance, if $ identifies some er- 
roneous state of the program, possibly $ holding almost certainly indicates 
a bug, even if the predicate was not true during this execution as seen by 
our omniscient observer. Further, for GP2, note that our specification does 
not include all cases in which the predicate holds, but only those cases de- 
tectable by observing the messages sent by each process in the system. In 
particular, if no communication takes place, no predicate will be detected. 

Additional uncertainty arises from the delays in monitoring, and effect of 
monitoring on the computation at hand. When detecting a global predicate, 
there is in general an unavoidable delay between local predicates becoming 
potentially true and the entire global predicate becoming true. Thus we 
can either detect the predicate in the past (GPl and GP2) or we must 
block the computation at critical points. How much do these restrictions 
confound the debugging process? The definitions GPl and GP2 imply that 
the predicate of interest may no longer be true when detection is reported, 
e.g. when the program is halted in order to examine local state. This is 
so if the predicate is not stable [1]. Thus one must either record relevant 
program state and include it in the messages sent to the monitoring process, 
or rely on some program replay or reversible execution technique to recover 
the state of interest. 

With definition GP3, we report the predicate when it is currently true 
in the global program state. However to achieve this we have to delay exe- 
cution of some processes. This is can be a serious impediment to debugging. 
By blocking some processes when the predicate becomes potentially true, 
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we may make the predicate either more or less likely to occur. For example, 
a predicate may be less likely to occur if processes “communicate” using 
timeouts or some other uncontrolled form of communication. The latter is 
a particular problem when processes are multithreaded; that is, consisting 
of multiple, independently schedulable threads of control which may com- 
municate through shared memory. It is rarely practical to monitor such 
communication when debugging without hardware or language support. 

2 Algorithms 

2.1 Detecting Possibly $ and Definitely $ 

The algorithms for detecting possibly $ and definitely $ are based on the 
same data structure: the lattice of consistent global states that correspond 
with an observed execution. Such a lattice consists of n orthogonal axes, 
with one axis for each monitored process. A point J = (x x ,x 2 , . . . ,x„) in 
this lattice corresponds to a global state in which process pi has taken x* 
steps. Of course, not all values of (xi,X 2 ,...,x n ) denote a point in the 
lattice, depending on the causal dependencies among the local states of P. 
Define the level of a point J to be the sum of its indices xi + x 2 + - . . + x„. 

Consider an observed behaviour of a system. A possible execution of 
this system is a total order of (consistent) global states in which exactly one 
process takes a step between adjacent global states. In terms of the lattice 
corresponding with the observed behavior, a possible execution is a path in 
this lattice where the level of each subsequent point in the path increases by 
one. A space- time diagram of a two-processes system and the corresponding 
lattice of global states is illustrated in Figure 2. A point Sij represents a 
state in which process pi is in its i th state and process p 2 is in its j th state. 
From the lattice, it is easy to see that one possible execution is the sequence 
of global states 

Si f \ ; S 2 ,i; 5 2t2 ; S 2 , 3; ^3,3; S 4t 3. . . 

For every point 2T in a lattice, there exists an execution that passes 
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through f. Hence, if any point in the lattice satisfies $, then by GP1 
possibly $ holds. The property definitely $ requires all possible executions 
to pass through a point that satisfies For example, in Figure 2 if the points 
{54,3, *£ 3 , 4 } both represent states that satisfy $ then definitely $ holds. This 
is because { 54 , 3 , 53 , 4 } are the only points with a level of 7 and all executions 
must include exactly one point with any given level. Definitely $ also holds 
if the states represented by points {5 5i3 , 5 3 i 5,5 6i 4,S5 j 5 ,54 |6 } all satisfy $. 
This is because if the execution does not pass through 55,3 or S 3|5 , then it 
must pass through 54,4 and hence through one of {Ss, 4, $5,5, S 4 i e}. 

Figures 3 and 4 give high-level algorithms used by the monitoring pro- 
cess po to detect possibly $ and definitely $ respectively. The definitely $ 
algorithm iteratively constructs the set of global states that have a level Ivl 
and are reachable from the initial state without passing through a global 
state that satisfies $. If there are no such states, then definitely $ holds by 
the level Ivl. 

In order to implement the algorithms in Figures 3 and 4, the monitored 
processes send their local states to the monitor po- Po maintains sequences 
of these states, one sequence per process, and assembles them into the nec- 
essary global states. The monitor must be able to determine when it can 
assemble ail the reachable global states of a given level and when it can drop 
a local state from its sequence because the local state will not appear in any 
further global states of interest. To achieve this, we use vector time stamps. 

Let Qi be the sequence of states that po maintains for p, stored in FIFO 
order. Each state e,' in Q, is labeled with a vector time stamp V(e,) where 
V(e, •)[»'] is the event number of p, that resulted in this state and V(e,)[_;'], j ^ 
i is the number of the latest event of pj that p,- is aware of. For example, in 
Figure 2, the vector timestamp of the fourth event of process pi is (4, 1). A 
set of local events ...,e„} with e, from process p, comprise a valid 

global state iff 


V*J : 1 < i,j < n : V(e,)[*] > V( ej )[i] 
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Intuitively, this condition states that a message cannot be received before it 
is sent. Note that the level of this global state is F(e u )[«]. 

For every event e, of process p, there exists a minimum global state 
S rain (ej) that contains e, and a maximum global state S max {ei ) that contains 
e,-. These global states are: 

Sm,n(e<) = (e u e 2 ,...,e n ) : V(ej)[j] = V( ei )[j] 

and 


‘S'maxC^i) — (^1* ^2j • • • » ^n) • 

< n«i)W A ve' : e, - e'j : V(e$)[i] > V(e.-)[i] 

These two global states limit the levels in which e j occurs. The minimum 
level containing e, is particularly easy to compute: it is the sum of compo- 
nents of the vector time stamp V(ej). So, po can construct the set of states 
with level Ivl when, for each sequence <?,, the sum of the components of 
the vector time stamp of the last element of Q, is at least Ivl. And, po can 
remove event e, from Q, when Ivl is greater than the level of the global state 

The two detection algorithms are linear in the number of global states, 
but unfortunately the number of global states is Sl(k n ) where k is the maxi- 
mum number events a monitored process has executed. There are techniques 
that can be used to limit the number of constructed global states. For exam- 
ple, a process p, need only send a message to po when p ( - potentially changes 
$ or when p, learns that pj has potentially changed $. Another technique 
is for pi to send an empty message to all other processes when p, potentially 
changes $. These, and other techniques for limiting the number of global 
states are discussed in [5]. 

2.2 Detecting Currently $ 

In contrast to the previous two algorithms, detecting Currently $ is com- 
putationally cheap but may block the monitored program, as discussed in 
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Section 1. Figures 5 and 6 give a protocol that detects Currently $ for 
conditions that do not reference the number of messages sent by a process 1 . 

This protocol is very simple; the main idea is to ensure that the condition 
holds long enough for the monitor to react. Whenever a process executes 
an event that potentially changes $, it sends its relevant state to p 0 , which 
maintains the latest received state for each process in P. If a process is 
about to execute an event that could make $ false, however, it first notifies 
Po and blocks before executing this event. The monitor then flushes all of 
the links into po» thereby examining all of the local process states of P up 
to the time that the first process blocked. If $ is not found to hold by the 
time all of the channels are flushed then po releases the blocked process. 

An example of the execution of this protocol is shown in Figure 7 in 
which the condition $ = Currently Xi + xj > 5 is being monitored. Each 
process can increment its value of x without blocking since doing so can 
only potentially make $ true. Process pi, however, before decrementing . 
value of x since doing so potentially makes false. The monitor therefore 
can consider the state ij = 2,xj — 2 before $ could become false. 

2.3 Comparison with Other Algorithms 

The work most similar in spirit to ours are the protocols developed by 
Spezialetti [8]. In particular, her event holding condition is the same spec- 
ification as our protocol for detecting Certainly $, and the specification of 
her event occurrence condition is similar to the specification of our Possibly 
$ algorithm. However, her protocols for non-local event detection are in- 
complete, in that they can miss conditions that in fact held. For example, 
the execution in Figure 8 shows such an execution. If the messages in tL 
figure correspond to the messages generated in establishing simultaneous re- 
gions [9], then her protocol will not detect Xi = x 2 , yet in fact the condition 
Definitely xj = x 3 holds. 

1 More precisely, i process’s local state with respect to 4 does not change when that 
process sends a message. 
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Our notion of possibly and definitely is closely related to the problem of 
access anomaly detection in parallel debugging and program analysis [10]. 
An important difference is that we wish to detect predicates over the states 
of the processes, while the access anomaly conditions are predicates over 
sequences of states (typically these are the sequences of instructions between 
synchronization events). Thus to apply our methods to access anomaly 
detection, we would have to augment process states with information on the 
read and write sets of global variables in a given instruction sequence. This 
is in fact how dynamic methods for detecting anomalies operate. 

Recent algorithms for dynamically detecting access anomalies have also 
employed vector timestamps [2,3]. The Flowback Analysis technique [2], 
computes before and after vectors for each synchronization event in a pro- 
gram execution. These are directly related to our minimum and maximum 
global states, S m in(e«) and S mai (e;), for an event e,. 

The Flowback Analysis technique first gathers traces of event sequences 
and performs determines event orderings off-line. Then various analyses are 
possible, of which access anomaly detection is just one. Our technique has 
two potential advantages: first, it does not require complete process traces 
to be recorded, and second, processes need not report all synchronization 
events, but only those that might change the predicate being detected. Of 
course the disadvantage is the the predicate must be specified before execu- 
tion of the program. 

3 Breakpoints and Tracepoints in Meta 

The Meta system [6] is a toolkit that provides the basic primitives needed 
to build a non-real-time reactive system. It finds application in areas such 
as distributed application management, performance monitoring, load bal- 
ancing, and distributed debugging. 

A Meta computation consists of a control program and the environment 
which it monitors and controls (see Figure 9). In the case of debugging, 
the control program is the distributed debugger, and the environment is the 
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distributed debuggee program. Using the Meta toolkit, the environment 
can be instrumented with sensors and actuators in order to expose its state. 
The control program monitors and controls the environment through the 
use of guarded commands (i.e. of the form when condition do action ) that 
reference the sensors and actuators of the instrumented environment. These 
guarded commands are compiled into a low-level postfix language that is 
executed by interpreters that reside in stubs co-located with the components 
of the distributed environment (in our case the processes of the debuggee 
program). This architecture facilitates fast notification and reaction. Each 
condition is a proposition on the state of system, and the action portion is a 
sequence of actuator invocations that is executed atomically. References to 
sensors or actuators may be both local (within the entity to which the stub 
is attached) or nonlocal, allowing one to write distributed control programs. 
The toolkit includes other features, for describing the system structure, fault 
tolerance and concurrency control, that do not concern us here. 

To make Meta useful for distributed debugging, we wish to instrument 
the debuggee program to provide the breakpoint and tracepoint facilities 
familiar to most programmers. In rule-based debuggers, breakpoints are 
often modeled as when condition do stop program , with the (often unstated) 
assumption that the state in which the program is stopped is (the first) one in 
which condition became true. As we have seen, when the condition is a global 
predicate, we must relax these requirements and detect the condition some 
time after it has happened, or perturb the computation being monitored. 
Also, since guarded commands are evaluated on the same computers as a 
the processes that they are monitoring, their impact on the performance of 
the debuggee is a concern. 

However there is a practical problem with the Meta implementation even 
with strictly local predicates. If a local predicate such as 

when in~procedure(proc3) do stop 

is implemented as an interpreted guarded command, there will be a small 
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but significant delay between detection of the condition and stopping the 
program. To avoid this we can model such a breakpoint as a modified kind 
of sensor that first halts the program, and then reports to the interpreter 
that the condition has become true and that the program is halted. In this 
view, Meta is monitoring a modified program that consists of the original 
program and the primitive breakpoints. We find utility in this concept for 
other applications of Meta. For instance when controlling a pressure vessel 
in an industrial automation project, one may prefer to use a pressure sensor 
that is directly connected to a safety shutoff valve. Such a sensor would 
report a potentially unsafe pressure reading after having shut off the valve, 
rather than relying on timely response from the Meta system to turn off the 
valve. 

We are now incorporating these algorithms into the Meta implementa- 
tion for use in debugging and other application areas. 
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Figure 2: An execution and the corresponding lattice of global states. 
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Possibly($): begin 

% Synchronize processes and distribute $ 

send $ to aii processes; 

current := global state S(0 t 0, . . M 0); 

release processes; 

lvl:= 0; 

do no state in current satisfies — * 

last := current 

W:= W+l; 

current ;= states of level hi reachable from a state in last ; 
od 
end; 

report Possibly $ 


Figure 3: Algorithm for detecting Possibly 
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Definitely^): begin 

% Synchronize processes and distribute 9 

send 9 to all processes; 

last := global state 5(0, 0, * . . , 0); 

release processes; 

remove all states in last that satisfy 9; 
hi := 1 ; 

% Invariant: last contains all states of level Ivl- 1 that are accessable 
% from 5(0, 0, ...» 0) without passing through a state satisfying 9. 
do last ^ { } — ► 

current states of level Ivl reachable from a state in last 
remove all states in current that satisfy 9; 

Ivl := lvl+ 1; last := current 
od 
end; 

report Definitely 9 


Figure 4: Algorithm for detecting Definitely 9. 


Currently($): begin 

M : array [l..n] of state := all empty; 
a: array [l..n] of integer := all 0; 
send $ to all processes; 
do M does not satisfy $ — ► 

receive message m from process 2 ; 
if m is a state message then M[i] := m 
else if m is a block message then 

o[»] := n- 1; 

send ack(i) to all processes except i 
else if m is an ack(j) message then 

a \j) *[j) - 1 : 

if a[j] = 0 then send unblock to process j; M [i] := empty; 
od 
end; 

report Currently $ 

Figure 5: Monitor protocol for detecting Currently 
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Client: cobegin 
receive 

for each event e that will potentially change $ do 
if e potentially rejects $ then 
send block to monitor; 
receive unblock; 

execute c and send state to monitor 
else execute c and send state to monitor 
od 

D 

do true — ► 

receive ack(j); 
send ack(.;) to monitor 
od 
coend 

Figure 6: Client protocol for detecting Currently 


xi := 2 x x := 1? (pi blocks) x\ := 1 



Figure 7: $ = (Currently X\ + > 5). 


17 


x\ := 4 ii := 3 



Figure 8: $ = (xi = z 2 ). 


actuators 



sensors 


Figure 9: The Meta computation model 
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